Skip to content

Tool-call gate (toolgate) and gateway scope enforcement - #59

Merged
parlakisik merged 5 commits into
mainfrom
feat/toolgate-and-gateway-scopes
Sep 6, 2026
Merged

parlakisik merged 5 commits into
mainfrom
feat/toolgate-and-gateway-scopes

Conversation

@fenar

@fenar fenar commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds the tool-call gate the Phase B roadmap lists as the governance piece, and fixes three authorization defects in aex-gateway that surfaced while putting the gateway in front of it.

Commit 1, toolgate. src/internal/toolgate decides every tool call on its argument values (scope per tool, then rules of kind ceiling, allowlist, lookup, sensitive_field, suffix, prefix; effect deny or escalate), records one nine-field artifact per call on a per-provider sha256 hash chain, publishes toolcall.requested|decided|executed|refused|escalated|approved into a new TOOLCALL JetStream stream, and settles held calls with an approver identity. src/aex-toolgate is the service form: an authorizing reverse proxy a provider runs in front of its tool endpoint (POST /v1/tools/{tool}, POST /v1/holds/{hash}, GET /v1/records[/verify]). hack/toolgate-harness, hack/ap-tools and hack/toolgate-client are the comparison harness, a mock provider and the client that replays a fixed call set against a live deployment and scores what each layer recorded. The event publisher now honours a caller-supplied idempotency_key (the old key, work_id plus the current second, deduplicated events without a work id published within one second).

Commit 2, gateway. Three defects, each with a regression test in src/aex-gateway/hack/tests/gateway_http_test.go:

  1. httpapi.NewRouter built an empty in-memory API-key table, so the documented identity-service validation was never used and every X-API-Key request got 401. The router now validates against IDENTITY_URL (API_KEY_VALIDATOR=memory keeps the empty table for tests).
  2. Scopes were validated, cached and carried in the request context, and nothing enforced them on any route. middleware.RequireScope now applies a route scope map after Auth: DefaultRouteScopes (reads need read, writes need the resource's scope) merged with ROUTE_SCOPES_FILE; * covers everything; unmapped routes require * (fail closed); refusals are 403 insufficient_scope with required_scope named.
  3. The gateway expected {"valid": true, ...} from /internal/v1/apikeys/validate and the identity service never sent valid. Identity now says valid: true; the gateway accepts a 200 with a tenant either way and refuses valid: false.

Also: the request log line could never carry request_id or tenant_id (the logging middleware ran outside the middleware that set them); a per-request holder fixes that and the line is now JSON through slog with scope_required and scope_decision. TOOLS_URL mounts /v1/tools/; the proxy forwards validated scopes as X-Scopes.

Review notes

  • DefaultRouteScopes is the one behavior change that touches every route: keys issued with custom scopes are now enforced, fail closed. Keys with the default ["*"] are unaffected. The scope names for existing routes (work:submit, bids:submit, contracts:write, providers:write, billing:write, tenants:write, certs:issue, read) are a proposal; please check them against how you issue keys today.
  • internal/toolgate/go.sum is intentionally empty (no dependencies) so the CI cache step finds a file.
  • Docs: docs/TOOLGATE.md (new), docs/AUTHENTICATION.md (route scopes), shared/schemas/events.md (tool-call events), mkdocs nav, Makefile and both CI matrices.

Test plan

  • go test ./... in internal/toolgate (11), aex-toolgate (3), aex-gateway (13, four new), aex-identity, internal/events
  • go build and go vet with default GOFLAGS in every touched module; go mod verify in aex-toolgate
  • Twelve-call comparison replayed on an OpenShift cluster: in memory, with the durable stream (72 of 72 events, chain verified), and through the live aex-identity (MongoDB), aex-gateway and aex-toolgate path (stream 72 to 104 events, decisions identical, the gateway's lines for a legitimate and a redirected payment identical apart from time and request id, the gate refusing the redirected one on its rule)
  • CI: lint-test and docker-build matrices now include internal/toolgate and aex-toolgate

…s and TOOLCALL stream

Adds the governance piece the Phase B roadmap lists: a gate that decides
every tool call an agent makes on the call's argument values, records one
artifact per call, and publishes the decision as events.

- src/internal/toolgate: the library. A provider policy (scope per tool,
  argument-value rules of kind ceiling, allowlist, lookup, sensitive_field,
  suffix, prefix; effect deny or escalate), a gate with ModeScope (tool
  name against scopes) and ModeFull (scope, then rules), Authorize,
  Resolve for held calls, a per-provider sha256 hash chain, VerifyChain,
  and Score (the five auditability dimensions of Nian et al., 2026).
  Call.Scopes lets a caller's validated scopes replace the policy grant.
- src/aex-toolgate: the service form, an authorizing reverse proxy a
  provider runs in front of its tool endpoint: POST /v1/tools/{tool},
  POST /v1/holds/{hash}, GET /v1/records and /v1/records/verify.
- src/internal/nats: TOOLCALL stream (toolcall.>, 365-day retention).
- src/internal/events: the publisher honours a caller-supplied
  idempotency_key; the old key (work_id plus the current second)
  deduplicated events without a work id published within one second.
- hack/toolgate-harness, hack/ap-tools, hack/toolgate-client: the
  comparison harness (three configurations over one fixed call set), the
  mock provider, and the client that replays the set against a live
  deployment and scores what each layer recorded.
- docs/TOOLGATE.md, shared/schemas/events.md, mkdocs nav, Makefile and CI
  matrices.

Tests: 11 in internal/toolgate, 3 in aex-toolgate/internal/httpapi.
Verified on an OpenShift cluster: the same twelve calls in memory, with
a durable JetStream record (72 of 72 events, chain verified) and through
the live gateway (104 events); decisions identical in every run.
…ructured request log

Three defects surfaced while putting the gateway in front of the
tool-call gate, each fixed with a regression test in
hack/tests/gateway_http_test.go:

1. httpapi.NewRouter built an empty in-memory API-key table, so the
   identity-service validation path (HTTPAPIKeyValidator) existed but
   was never used and every X-API-Key request got 401. The router now
   validates against IDENTITY_URL; API_KEY_VALIDATOR=memory keeps the
   empty table for tests.
2. Scopes were validated, cached and placed in the request context, and
   nothing checked them against a route. middleware.RequireScope now
   enforces a route scope map after Auth: DefaultRouteScopes (reads need
   "read", writes need the resource's scope) merged with ROUTE_SCOPES_FILE
   ({"METHOD /prefix": "scope"}); "*" covers everything; an unmapped route
   requires "*" (fail closed). Refusals are 403 insufficient_scope with
   required_scope named. Keys created with the default ["*"] are
   unaffected.
3. The gateway expected {"valid": true, ...} from
   /internal/v1/apikeys/validate and the identity service never sent a
   "valid" field, so even a wired validator would have refused every key.
   Identity now says valid: true; the gateway accepts a 200 with a tenant
   whether or not the field is present and refuses valid: false.

Also: the request log line could never carry request_id or tenant_id
because Logging ran outside the middleware that set them; a per-request
holder (RequestInfo) fixes that, and the line is now a JSON record
through slog with scope_required and scope_decision. TOOLS_URL mounts
/v1/tools/ for a provider's tool endpoint, and the proxy forwards the
validated scopes as X-Scopes so a downstream gate can check per tool.
…thorization

Six ways through the gate, each with a regression test. The first four were
verified against testdata/policy.json before the fix: all returned allow.

1. A ceiling compares numbers, and toFloat only handles numeric kinds, so a
   quoted amount made the rule not fire: send_payment with {"amount":"999999"}
   passed the 5000 ceiling as a clean scope-only allow. A value that cannot be
   compared is now a deny.
2. Rule.evaluate returned "not fired" when the constrained argument was absent,
   so any tool whose upstream accepts a synonym, an alternate casing or its own
   default walked past ceiling, allowlist, suffix, prefix and sensitive_field:
   send_email with {"recipient": ...} was allowed where {"to": ...} is denied.
   The guard now fails closed, as evaluateLookup already did for the same case.
3. Call.Scopes replaced the policy's granted_scopes outright, and the value
   comes from the X-Scopes header, which the gated agent controls. Sending
   "*" switched the tool_scopes layer off: delete_invoice, whose scope the
   policy does not grant, was allowed. Scopes now intersect, so a caller can
   narrow the grant and never widen it.
4. DefaultRouteScopes had no /v1/tools/ entry, so tool calls fell through to
   the "*" default; the gateway then forwarded X-Scopes: "*", which is the one
   value that neutralised the gate's own check. With (3) the layers no longer
   cancel, and POST /v1/tools now maps to tools:invoke.
5. POST /v1/holds/{hash} was unauthenticated and took the approver's name from
   the request body, so an agent replayed the hold hash it was handed in the
   202 and approved its own bank-account change, with the fabricated name
   recorded as human approval. GET /v1/records was also unauthenticated and
   served Args verbatim: payment amounts and recipient accounts to anyone who
   could reach the pod. Both now need OPERATOR_TOKEN, and are refused rather
   than open when none is set. Argument values are off the records response
   unless EXPOSE_ARGS=true.
6. Route prefixes matched with HasPrefix and no segment boundary, so a sibling
   silently inherited a shorter route's scope: /v1/tools/list_invoices covered
   /v1/tools/list_invoices_and_delete, and /v1/work would cover a later
   /v1/workflows. Under-protection, not a 403, so nothing surfaced it.

Also, on the audit record the component exists to produce:

- Arguments were round-tripped through float64, so an account number past 2^53
  reached the tool as a different number and the artifact recorded the changed
  one. Decoding uses UseNumber; toFloat already handled json.Number.
- publish errors were discarded, so a tool ran while its event was dropped and
  only the in-memory chain survived. Failures are logged and counted, and
  /v1/records/verify reports the count.
- The chain is process-local, so a restart starts again at genesis and
  VerifyChain still answers ok: a fresh chain and a truncated one looked the
  same. Each gate now has a chain_id, reported by verify, so the discontinuity
  is visible. Persisting the chain is still open.
- The image baked internal/toolgate/testdata/policy.json in as the default
  POLICY_FILE, so a deployment that forgot its ConfigMap authorized real calls
  against the fixture, which grants payments:send. No policy is baked in and
  the gate refuses to start without one.

docs/TOOLGATE.md states the trust boundary the code now enforces: identity
headers are attribution, the operator endpoints are not part of that model.
docs/AUTHENTICATION.md notes that HTTPAPIKeyValidator caches a validation for
five minutes with no invalidation, so a revoked key keeps working until it
expires.

Tests: 8 in internal/toolgate, 4 in aex-toolgate/internal/httpapi, 2 in
aex-gateway/internal/middleware, 3 cases in the gateway's scope table.
CVE-2026-56854 (CRITICAL, golang.org/x/crypto/ssh) is fixed repo-wide on main
by #60. aex-toolgate and hack/toolgate-harness are new here, so they do not
exist on that branch and pinned v0.52.0 of their own.

Same reasoning as #60: the path is not reachable (AEX opens no SSH connection;
x/crypto arrives under the nats and mongo drivers) and govulncheck passes. The
bump keeps these two modules level with the rest of the repo after the rebase.
@parlakisik
parlakisik force-pushed the feat/toolgate-and-gateway-scopes branch from 38e4710 to 7d0f7fb Compare September 6, 2026 02:22
…sions legible

A team review of the previous commit found that it closed the argument-value
bypasses for five rule kinds and missed the sixth, and that two of its own
changes were wrong. All findings were reproduced before fixing and each fix has
a test that fails without it.

Rule engine:

- The lookup rule still failed open when the argument it checks was absent, so
  send_payment with "recipientAccount" (or with no recipient at all, letting
  the tool default it) skipped the rule that says where money may go and was
  allowed. evaluateLookup now fails closed like the other kinds. The previous
  commit's message claimed it already did; it did not.
- sensitive_field compared with exact fmt.Sprint equality, so "Bank_Account",
  "BANK_ACCOUNT", "bank-account", a padded value or ["bank_account"] all
  missed the rule and skipped the human approval entirely. Values are
  normalised (case, surrounding space, hyphen for underscore) and a value that
  is not a lone scalar fails closed.
- suffix and prefix matched anywhere in the value, so
  "attacker@evil.example,someone@corp.example" passed the allowed-domain check
  and "corp-internal://../../../s3://attacker/" passed the destination check.
  A value carrying separators, padding or control characters is no longer a
  single value and fails closed; a prefix cannot be escaped with "..".
- A rule that fires because the gate could not check it now reports
  "cannot verify: <detail>" instead of borrowing the rule's own message. An
  approver looking at a vendor rename was being told "bank account changes
  need a person".

Scopes:

- narrow() intersected the policy's grant with the caller's, which denied
  every call when the policy grants "*": the intersection was empty. A
  wildcard policy now takes the caller's scopes as the narrower set.
- The gateway and the gate use different scope vocabularies, so a key needs
  "tools:invoke" to pass the route check AND the policy's per-tool scopes to
  pass the gate. Nothing said so and nothing tested it, so the first thing an
  operator provisions looks broken. Documented, tested, and the rationale in
  scopes.go corrected: it claimed "*" neutralised the downstream gate, which
  narrow() had already stopped being true in the same commit.
- An entry with an empty path in ROUTE_SCOPES_FILE matched every path and
  outranked the fail-closed default, opening the whole unmapped surface.

Records:

- ToEventData decoded through float64, so the published event -- the only copy
  that survives a restart -- rounded integers past 2^53 that the in-memory
  record held exactly.
- GET /v1/records redacted Args but served Outcome, which quotes 120 characters
  of the tool's response; for a read-shaped tool that is where the account
  numbers are.
- records and held grew without bound under agent-driven load. Capped at
  MaxRecords/MaxHeld, with records_dropped reported by /v1/records/verify.
- requireOperator used TrimPrefix, a no-op when the prefix is absent, so a bare
  token with no Bearer scheme authenticated.

Tests: 6 new cases in internal/toolgate. Each was confirmed to fail against the
unfixed code in a scratch copy.
@parlakisik
parlakisik merged commit dffae25 into main Sep 6, 2026
70 checks passed
@parlakisik
parlakisik deleted the feat/toolgate-and-gateway-scopes branch September 6, 2026 03:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants